Merge branch 'main' into add_fabric_warehouse

Author: fresioAS

.circleci/continue_config.yml

@@ -306,6 +306,7 @@ workflows:
                 - "3.10"
                 - "3.11"
                 - "3.12"
+                - "3.13"
       - cicd_tests_windows
       - engine_tests_docker:
           name: engine_<< matrix.engine >>

.github/workflows/pr.yaml

@@ -26,7 +26,8 @@ jobs:
       - name: Run CI
         run: pnpm run ci
   test-vscode-e2e:
-    runs-on: ubuntu-latest
+    runs-on:
+      labels: [ubuntu-2204-8]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
@@ -41,22 +42,43 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: '3.12'
+      - name: Cache pip dependencies
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key:
+            ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt',
+            '**/pyproject.toml', '**/setup.py') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+      - name: Cache virtual environment
+        uses: actions/cache@v4
+        with:
+          path: .venv
+          key:
+            ${{ runner.os }}-venv-${{ hashFiles('**/requirements*.txt',
+            '**/pyproject.toml', '**/setup.py') }}
+          restore-keys: |
+            ${{ runner.os }}-venv-
       - name: Install python dependencies
         run: |
           python -m venv .venv
           source .venv/bin/activate
           make install-dev
-      - name: Fetch VS Code
-        working-directory: ./vscode/extension
-        run: pnpm run fetch-vscode
-
       - name: Install code-server
         run: curl -fsSL https://code-server.dev/install.sh | sh
       - name: Install Playwright browsers
         working-directory: ./vscode/extension
         run: pnpm exec playwright install
       - name: Run e2e tests
         working-directory: ./vscode/extension
+        timeout-minutes: 90
         run: |
           source ../../.venv/bin/activate
-          pnpm run test:e2e tests/stop.spec.ts
+          pnpm run test:e2e
+      - uses: actions/upload-artifact@v4
+        if: ${{ !cancelled() }}
+        with:
+          name: playwright-report
+          path: vscode/extension/playwright-report/
+          retention-days: 30

.github/workflows/release_extension.yaml

@@ -32,7 +32,7 @@ jobs:
         with:
           node-version: '20'
       - name: Install pnpm
-        uses: pnpm/action-setup@v2
+        uses: pnpm/action-setup@v4
         with:
           version: 10
       - name: Install dependencies

.prettierignore

@@ -20,6 +20,9 @@ vscode/extension/out
 vscode/extension/src_react
 vscode/extension/tsconfig.tsbuildinfo
 vscode/extension/.vscode-test/
+vscode/extension/playwright-report/
+vscode/extension/test-results/
+vscode/extension/.test_setup
 
 sqlmesh
 docs

docs/concepts/macros/macro_variables.md

@@ -68,6 +68,7 @@ Prefixes:
 Postfixes:
 
 * dt - A python datetime object that converts into a native SQL `TIMESTAMP` (or SQL engine equivalent)
+* dtntz - A python datetime object that converts into a native SQL `TIMESTAMP WITHOUT TIME ZONE` (or SQL engine equivalent)
 * date - A python date object that converts into a native SQL `DATE`
 * ds - A date string with the format: '%Y-%m-%d'
 * ts - An ISO 8601 datetime formatted string: '%Y-%m-%d %H:%M:%S'
@@ -83,6 +84,11 @@ All predefined temporal macro variables:
     * @end_dt
     * @execution_dt
 
+* dtntz
+    * @start_dtntz
+    * @end_dtntz
+    * @execution_dtntz
+
 * date
     * @start_date
     * @end_date

docs/concepts/models/model_kinds.md

@@ -935,7 +935,13 @@ SQLMesh achieves this by adding a `valid_from` and `valid_to` column to your mod
 
 Therefore, you can use these models to not only tell you what the latest value is for a given record but also what the values were anytime in the past. Note that maintaining this history does come at a cost of increased storage and compute and this may not be a good fit for sources that change frequently since the history could get very large.
 
-**Note**: Partial data [restatement](../plans.md#restatement-plans) is not supported for this model kind, which means that the entire table will be recreated from scratch if restated. This may lead to data loss, so data restatement is disabled for models of this kind by default.
+**Note**: SCD Type 2 models support [restatements](../plans.md#restatement-plans) with specific limitations:
+
+- **Full restatements**: The entire table will be recreated from scratch when no start date is specified
+- **Partial restatements**: You can specify a start date to restate data from a certain point onwards to the latest interval. The end date will always be set to the latest interval's end date, regardless of what end date you specify
+- **Partial sections**: Restatements of specific sections (discontinued ranges) of the table are not supported
+
+Data restatement is disabled for models of this kind by default (`disable_restatement true`). To enable restatements, set `disable_restatement false` in your model configuration.
 
 There are two ways to tracking changes: By Time (Recommended) or By Column.
 
@@ -1283,11 +1289,11 @@ This is the most accurate representation of the menu based on the source data pr
 
 ### Processing Source Table with Historical Data
 
-The most common case for SCD Type 2 is creating history for a table that it doesn't have it already. 
+The most common case for SCD Type 2 is creating history for a table that it doesn't have it already.
 In the example of the restaurant menu, the menu just tells you what is offered right now, but you want to know what was offered over time.
 In this case, the default setting of `None` for `batch_size` is the best option.
 
-Another use case though is processing a source table that already has history in it. 
+Another use case though is processing a source table that already has history in it.
 A common example of this is a "daily snapshot" table that is created by a source system that takes a snapshot of the data at the end of each day.
 If your source table has historical records, like a "daily snapshot" table, then set `batch_size` to `1` to process each interval (each day if a `@daily` cron) in sequential order.
 That way the historical records will be properly captured in the SCD Type 2 table.
@@ -1433,11 +1439,14 @@ GROUP BY
   id
 ```
 
-### Reset SCD Type 2 Model (clearing history)
+### SCD Type 2 Restatements
 
 SCD Type 2 models are designed by default to protect the data that has been captured because it is not possible to recreate the history once it has been lost.
 However, there are cases where you may want to clear the history and start fresh.
-For this use use case you will want to start by setting `disable_restatement` to `false` in the model definition.
+
+#### Enabling Restatements
+
+To enable restatements for an SCD Type 2 model, set `disable_restatement` to `false` in the model definition:
 
 ```sql linenums="1" hl_lines="5"
 MODEL (
@@ -1449,16 +1458,39 @@ MODEL (
 );
 ```
 
-Plan/apply this change to production.
-Then you will want to [restate the model](../plans.md#restatement-plans).
+#### Full Restatements (Clearing All History)
+
+To clear all history and recreate the entire table from scratch:
 
 ```bash
 sqlmesh plan --restate-model db.menu_items
 ```
 
 !!! warning
 
-    This will remove the historical data on the model which in most situations cannot be recovered.
+    This will remove **all** historical data on the model which in most situations cannot be recovered.
+
+#### Partial Restatements (From a Specific Date)
+
+You can restate data from a specific start date onwards. This will:
+- Delete all records with `valid_from >= start_date`
+- Reprocess the data from the start date to the latest interval
+
+```bash
+sqlmesh plan --restate-model db.menu_items --start "2023-01-15"
+```
+
+!!! note
+
+    If you specify an end date for SCD Type 2 restatements, it will be ignored and automatically set to the latest interval's end date.
+
+```bash
+# This end date will be ignored and set to the latest interval
+sqlmesh plan --restate-model db.menu_items --start "2023-01-15" --end "2023-01-20"
+```
+
+
+#### Re-enabling Protection
 
 Once complete you will want to remove `disable_restatement` on the model definition which will set it back to `true` and prevent accidental data loss.
 

docs/guides/configuration.md

@@ -98,7 +98,52 @@ All software runs within a system environment that stores information as "enviro
 
 SQLMesh can access environment variables during configuration, which enables approaches like storing passwords/secrets outside the configuration file and changing configuration parameters dynamically based on which user is running SQLMesh.
 
-You can use environment variables in two ways: specifying them in the configuration file or creating properly named variables to override configuration file values.
+You can specify environment variables in the configuration file or by storing them in a `.env` file.
+
+### .env files
+
+SQLMesh automatically loads environment variables from a `.env` file in your project directory. This provides a convenient way to manage environment variables without having to set them in your shell.
+
+Create a `.env` file in your project root with key-value pairs:
+
+```bash
+# .env file
+SNOWFLAKE_PW=my_secret_password
+S3_BUCKET=s3://my-data-bucket/warehouse
+DATABASE_URL=postgresql://user:pass@localhost/db
+
+# Override specific SQLMesh configuration values
+SQLMESH__DEFAULT_GATEWAY=production
+SQLMESH__MODEL_DEFAULTS__DIALECT=snowflake
+```
+
+See the [overrides](#overrides) section for a detailed explanation of how these are defined.
+
+The rest of the `.env` file variables can be used in your configuration files with `{{ env_var('VARIABLE_NAME') }}` syntax in YAML or accessed via `os.environ['VARIABLE_NAME']` in Python.
+
+#### Custom dot env file location and name
+
+By default, SQLMesh loads `.env` files from each project directory. However, you can specify a custom path using the `--dotenv` CLI flag directly when running a command:
+
+```bash
+sqlmesh --dotenv /path/to/custom/.env plan
+```
+
+!!! note
+    The `--dotenv` flag is a global option and must be placed **before** the subcommand (e.g. `plan`, `run`), not after.
+
+Alternatively, you can export the `SQLMESH_DOTENV_PATH` environment variable once, to persist a custom path across all subsequent commands in your shell session:
+
+```bash
+export SQLMESH_DOTENV_PATH=/path/to/custom/.custom_env
+sqlmesh plan
+sqlmesh run
+```
+
+**Important considerations:**
+- Add `.env` to your `.gitignore` file to avoid committing sensitive information
+- SQLMesh will only load the `.env` file if it exists in the project directory (unless a custom path is specified)
+- When using a custom path, that specific file takes precedence over any `.env` file in the project directory.
 
 ### Configuration file
 

docs/guides/projects.md

@@ -27,18 +27,18 @@ To create a project from the command line, follow these steps:
 1. To scaffold a project, it is recommended that you use a python virtual environment by running the following commands:
 
     ```bash
-    python -m venv .env
+    python -m venv .venv
     ```
 
     ```bash
-    source .env/bin/activate
+    source .venv/bin/activate
     ```
 
     ```bash
     pip install sqlmesh
     ```
 
-    **Note:** When using a python virtual environment, you must ensure that it is activated first. You should see `(.env)` in your command line; if you don't, run `source .env/bin/activate` from your project directory to activate your environment.
+    **Note:** When using a python virtual environment, you must ensure that it is activated first. You should see `(.venv)` in your command line; if you don't, run `source .venv/bin/activate` from your project directory to activate your environment.
 
 1. Once you have activated your environment, run the following command and SQLMesh will build out your project:
 

docs/guides/ui.md

@@ -24,7 +24,7 @@ For development work, we recommend using the SQLMesh UI alongside an IDE. The UI
 
 Before beginning, ensure that you meet all the [prerequisites](../prerequisites.md) for using SQLMesh. The SQLMesh browser UI requires additional Python libraries not included in the base SQLMesh installation.
 
-To use the UI, install SQLMesh with the `web` add-on. First, if using a python virtual environment, ensure it's activated by running `source .env/bin/activate` command from the folder used during [installation](../installation.md).
+To use the UI, install SQLMesh with the `web` add-on. First, if using a python virtual environment, ensure it's activated by running `source .venv/bin/activate` command from the folder used during [installation](../installation.md).
 
 Next, install the UI with `pip`:
 

docs/installation.md

@@ -8,12 +8,12 @@ It is recommended, but not required, that you use a python virtual environment w
 
 First, create the virtual environment:
 ```bash
-python -m venv .env
+python -m venv .venv
 ```
 
 Then activate it:
 ```bash
-source .env/bin/activate
+source .venv/bin/activate
 ```
 
 ## Install SQLMesh core